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TRACING TECHNIQUE AND RECORDING MEDIA 
OF OBJECT MOTION 



BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to an object motion tracing technique for 
tracing the three-dimensional motion of an object obtained in time series on 
the basis of the stereo image of the object concerned, and also to a recording 
media wherein an executing program for object motion tracing is recorded. 

2. Brief Description of the Prior Art 

Generally, for tracing the three-dimensional rigid body movement of a 
curve-faced object, time-series images consecutively obtained from a visual 
sensor as an image pick-up means are used. If the form of the traced object is 
known, the tracing point can be estimated in correspondence search by using a 
three-dimensional geometric model of the object concerned, thereby realizing 
highly- accurate tracing of the position and the posture of the curve-faced 
object, wherein tracing is comparatively unaffected by partial occlusion. 

Prior arts known for tracing the three-dimensional rigid body motion 
of an object using an object model are as follows: 

(1) Method of using a monocular CCD camera as a sensor, 

(2) Method of applying a stereo camera system consisting of a plural 
number of CCD cameras as sensors, and 

(3) Method of using a range finder as a sensor. 

The silhouette of the curve-faced object as mentioned above is an 
apparent silhouette whose shape consecutively varies according to the 
position/posture of the object concerned, and it is difficult to estimate and 
measure its position accurately. Therefore, the objects traced by the above- 
mentioned prior arts (1) or (2) were limited to those having fixed-edge 
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structures that are invariable according to observing directions, such as 
polyhedrons, cylinders, etc., but arbitrary-curve-faced objects with smooth 
faces have never been traced by those prior arts. In the prior art (3) above, 
objects of free forms including arbitrary-curve -faced objects can be traced by 
utilizing the three-dimensional information on the surface, but a special type 
of sensor such as a high-speed laser range finder is needed for that purpose. 

SUMMARY OF THE INVENTION 

The present invention was carried out in consideration of the aforesaid 
problems, and aims at providing an object motion tracing technique whereby 
the three-dimensional movement of a curve-faced object is traced accurately 
at a high speed, utilizing an object model and applying as sensors a stereo 
camera system consisting of a group of image pick-up cameras. 

The object motion tracing technique related to the present invention is 
a method of tracing the three-dimensional motion of an object, based on the 
stereo image of the object concerned obtained in time series, composed of a 
selection step for selecting a tracing point corresponding to the silhouette of 
the object concerned on the stereo image of aforesaid object, a sampling step 
for sampling from the stereo image corresponding points on the silhouette of 
the object corresponding to respective tracing points; a measuring step for 
measuring the three-dimensional coordinates of the sampled corresponding 
points, and a detecting step for detecting the position/posture of the object 
concerned from each of the three-dimensional coordinates of the tracing points 
and the corresponding points, wherein the three-dimensional motion of the 
object can be traced by continuously repeating each process on above- 
mentioned steps from the selecting step through the detecting toward each 
frame of the stereo image obtained in time series. 

A recording media related to the present invention is a media wherein an 
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object motion tracing program is recorded for tracing the three-dimensional 
movement of an object based on the stereo image of the object obtained in time 
series, composed of a selecting step for selecting tracing points corresponding 
to the silhouette of the object concerned on the stereo image of the object, a 
sampling step for sampling from the stereo image the corresponding points on 
the silhouette of the object corresponding to each tracing point; a measuring 
step for measuring the three-dimensional coordinates of the sampled 
corresponding points, and a detecting step for detecting the position/posture 
of the object from each of the three-dimensional coordinates of the tracing 
points and the corresponding points, wherein a program is recorded for 
executing the tracing of the three-dimensional motion of the object by 
continuously repeating each process on those steps from the selecting step 
through the detecting one toward each frame of the stereo images obtained in 
time-series. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG.l is a block diagram showing a system configuration for embodying the 
present invention. 

FIG. 2 is a flow chart showing the flow of basic processing of the present 
invention, 

FIG. 3 is an initial picture of time-series stereo image. 

FIG. 4 is an example of the initial frame of a curve-faced object. 

FIG. 5 is an example of the net configuration of a curve-faced object model. 

FIG. 6 is a method of searching for a corresponding edge point. 

FIG. 7 is a method of searching for a stereo corresponding point, 

FIG. 8 is a method of three-dimensional estimation of a corresponding point by 

monocular vision, 

FIG. 9 is a measurement error of a curve-faced object in stereo vision. 
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FIG. 10 is an example of a tracing result of an arbitrarily-curved object „ 
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

Hereinafter the object motion tracing technique of the present invention 
is described in detail based on the embodiment.. 

FIG. 1 is a block diagram showing the system configuration for executing 
the method of generating a three-dimensional object model by the present 
invention. In this Figure, a numerical sign 1 is a computer to control the 
entire system, and is connected with each of the under-mentioned parts 
through data bus 2. 

Signs 3a, 3b and 3c stand for television cameras, whose analog output 
data are transmitted to data bus 2 after being converted to digital signals by 
A/D Converters 4a, 4b and 4c. 

A numerical sign 5 stands for the image memory for storing image data 
on the object picked up by television cameras 3a, 3b and 3c, while 6 stands for 
the display unit for displaying the image, numeral 7 shows a printer and 8 a 
keyboard terminal. 

Numeral 9 shows a hard disk unit for storing the system software for the 
object motion tracing program related to the present invention and storing the 
three-dimensional geometric model of the object concerned, as well as for 
consecutively recording the three-dimensional information restored by stereo 
vision and the information on the object motion parameter obtained by the 
motion tracing process. 

Numeral 10 stands for the CD-ROM Drive for reading CD-ROMs wherein 
the object motion tracing program is recorded, and the above-mentioned 
system software is installed onto the Hard Disk 9 from this CD-ROM. 

The aforementioned system is connected to an external host computer, 
etc. via data bus 2. 
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FIG. 2 is a flow chart of the basic processing operation in the system of 
aforesaid configuration, showing the processing flow for tracing the three- 
dimensional rigid body motion of a curve-faced object on the basis of the stereo 
image of the object concerned obtained in time-series. 

This processing operation is carried out by a program pre -memorized by 
the Computer 1 in FIG. 1, consisting of: 

the input step (Si) where the stereo image of an arbitrary-curve-faced 
object performing a three-dimensional rigid body motion is inputted as a 
frame signal from an image pick-up camera applying CCD, etc.; 

the selecting step (S2) where the tracing points corresponding to the 
silhouette of an object on the stereo image of the object concerned is selected; 

the sampling (searching) step (S3) where the corresponding point on the 
silhouette of the object corresponding to each tracing point is sampled from 
above-mentioned stereo image.; 

the measuring step (S4) where the three-dimensional coordinates of the 
sampled corresponding points are measured (restored); 

the detecting step (S5) where the position/posture of the above- 
mentioned object is detected from each of the three-dimensional coordinates of 
the tracing points and the corresponding points; and 

the discriminating step (S6) where it is discriminated whether the error 
of measured position/posture of the object concerned is small enough or not, 
wherein; 

the three-dimensional motion of the object is traced by continuously 
repeating the processing (S2 ~ S5) on each of those steps from the selecting 
step (S2) through the detecting one (S5) toward each frame of the stereo image 
obtained in aforementioned time series. 

Firstly, a stereo image is inputted (SI), and the tracing points 
corresponding to the silhouette of a curve-faced object observed on the stereo 
image are selected based on information on the present position of the object 
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concerned, using a three-dimensional geometric model (S2). Then, the 
corresponding points on the silhouette of the object corresponding to 
respective tracing points are sampled from aforesaid stereo image (S3), so 
that their three-dimensional coordinates are measured. (S4), 

Then, based on the group of the three-dimensional coordinates of 
respective tracing points and respective measured corresponding points, the 
position/posture of the object concerned and the error are determined (S5), 
and it is discriminated whether the error is small enough or not (S6). In case 
the error is not small enough, then the processes from (S2) above are repeated, 
regarding the detected position/posture as the renewed present position of the 
object concerned. If the errors are small enough, the same processes from 
(SI) above are repeated, regarding the detected position/posture of the object 
concerned as the renewed present position. 

Hereunder the above processes are described in detail, referring to FIG. 
3 through FIG. 10. 

FIG. 3 shows an example of the initial frame (frame 0) of an inputted 
time-series stereo image. For tracing the motion of an object using a 
geometric model of the object concerned, it is essential to accurately 
determine, as the initial tracing position, the three-dimensional 
position/posture of the object concerned in the initial frame. The description 
of the method of estimating the initial position of an object is omitted, as it is 
not necessary here. 

FIG. 4 is a drawing showing an example of estimated initial position of 
an arbitrary-curve-faced object (a toy banana here), together with the three- 
dimensional silhouette in a scene restored by the segment-based stereo 
method. 

Following is the description of how the tracing points corresponding to 
the silhouette of a curved object are selected from the object model. 

FIG. 5 is an example of net configuration of an object model. Here, an 



6 



arbitrarily-curved object is modeled as a net configuration with nodes of a 
limited number of representative points (model points) having a spatially 
neighboring mutual relationship (four couplings). 

The silhouette of an arbitrarily-curved object is an apparent silhouette 
which consecutively transforms according to observing directions. For 
tracing the motion of an arbitrarily-curved object, the rigid body motion 
parameters of an object between frames of inputted images in time series and 
the form of the silhouette varying every moment need to be estimated at the 
same time. 

For that purpose, the form of the hidden silhouette is roughly estimated 
at a high speed on the basis of the assumed position/posture of an object in a 
frame, using an object model and the known observing position. It can be 
discriminated by the following formula whether or not a point on the object 
surface corresponding to the model point P is observable from the observing 
the position O at the assumed position of the targeted object. 
Formula (1) 



In this formula, N is a normal vector at a point P. Therefore, the hidden 
silhouette of a curved surface observed on the image can be regarded as the 
boundary of an area consisting of observable model points, or, it can be 
regarded to be equal to P in the following formula. 

(Formula 2) 




(1) 




(2) 
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In this formula, P next(i) represents a model point adjacent to P. A group 
of such model points {PJ can be fetched by the following processes (a) and (b) 
towards the net configuration of the model. 

(a) The nodes satisfying the formula (1) are labeled. 

(b) The nodes satisfying the formula (2) are searched out, and starting 
from the searched node, other nodes satisfying the formula (2) are traced. 

These processes can be carried out at a high speed by a simple raster 
scan. Hidden portions are not checked here, as it is costly to do so. 

Motion between frames is estimated by searching for the corresponding 
points on a following frame image, regarding a number of model points 
obtained by above-mentioned processing as tracing points. Though three- 
dimensional movement can be estimated in principle if there are three points 
that are not on a same straight line, as many tracing points as processing time 
allows are used here in order to obtain strength against occlusion. 

Hereunder the searching procedure for corresponding points is described. 
Object motion tracing is the repetition of the process for determining the 
corresponding points in the frame ( F r a m e (t + 1) ) at a following time t + 1 
corresponding to the tracing point P at a time t, and can be carried out by 
processes (c) and (d) below. 

(c) Search for a Corresponding Edge Point 

From the position P L where P is projected into the initial image 
Frame L , pixel is vertically traced to search for the maximal point 
D L of the differentiated value of pixel. 

(d) Search for a Stereo Corresponding Point 

The epipolar line on image Frame R corresponding to D L is 
determined, and as shown in FIG. 7, D R is searched for in the epipolar 
line direction as in FIG. 6, from the point P' R which is the nearest 
point to the tracing point P R on the epipolar line. 
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These processes are basically differentiation processes for the simple 
one -dimensional data obtained by tracing, and the calculation cost is small. 

Hereunder the restoration of the three-dimensional coordinates of 
corresponding points by the stereo method is described. The coordinate 
values of {D L , D K } searched for by above-mentioned processes are converted to 
sub pixel based on the inclination of neighboring pixel values, to determine 
the three-dimensional coordinate D on the basis of the principle of 
trigonometrical survey. 

Generally, when the edge direction of a tracing point is parallel or almost 
parallel to the epipolar line, binocular stereo measurement causes a bigger 
error. The easiest way to solve this is to apply a trinocular stereo system 
with an extra camera Frame v above or below the baseline of the right and 
the left stereo cameras. For each of the tracing points, out of the trinocular 
combinations {L, R}, {L, V) and {R, V}, a combination with a larger angle 
formed by the edge direction and the epipolar line is selected to execute the 
searching process as above-mentioned. Then, for the obtained corresponding 
point {Dj, miscorresponding is eliminated by checking the distance I Pj - Dj 
from the model point. 

Following is the description of restoration of the three-dimensional 
coordinates of corresponding point by monocular vision. Search for 
corresponding points does not always proceed successfully due to occlusion 
and/or noise. If a corresponding point can be found only monocularly, or only 
by a binocular combination which is almost parallel to epipolar line, 
miscorresponding data are eliminated as follows: As shown in FIG. 8, when 
the end of the vertical line starting from P to the projected line connecting the 
centers of the camera lenses C and D is regarded as D', miscorresponding data 
are eliminated for the obtained corresponding point { D s } by checking the 
distance I P t — Di I from a model point. 
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Then, description is made on the method of determining the 
position/posture and error of the object concerned on the basis of the group of 
the three-dimensional coordinates of respective measured corresponding 
points and tracing points. The motion parameter of an object from Frame 
(t) to Frame (t + 1) can be expressed by 3x3 rotation matrix R and translation 
vector T, and can be calculated by the least square method in the following 
formula which minimizes the error in moving { to {D}. 

min S I RP i+T — D i I 2 (3) • 

R, T i=l 

Here, the position/posture of the model is renewed according to 
calculated R and T. If the error is not small enough, the processes starting 
from estimation of the hidden silhouette are repeated to make the error 
convergent by so-called "hill-climbing method". As well known, in measuring 
the hidden silhouette of a curved surface by stereo vision, the error as shown 
in FIG. 9 is included. When the baseline length of cameras is 1, the distance 
to the object is d and the curvature of the curving surface concerned is 1 / r, 
the size of the error e between' the hidden silhouette estimated from the model 
and the value measured by stereo vision may be regarded as e^r 1 / 2 d, in 
case of 1 < d and r < d. As e is regarded to be small enough in many cases, it 
is not taken into account here. 

FIG. 10 shows an example of the result of motion tracing experiment 
using a rotary table for an arbitrarily-curved object model shown in FIG. 3 
(Diameter: about 3cm, Space between nodes: about 3mm). The model is 
moved according to R and T determined in each frame, so that the model nodes 
are projected onto the inputted image. The experiment proves that tracing is 
possible under a partial-occlusion environment. 
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Thus, this embodiment can realize high-speed and accurate three- 
dimensional rigid body motion tracing of a curved object utilizing an object 
model, by applying a stereo camera system consisting of a group of CCD 
cameras as sensors. 

Further, the program for executing the above-mentioned three- 
dimensional rigid body motion tracing of a curved object can be recorded in an 
optional recording media, in the form readable by a computer. 

As described above, according to the present invention, the three- 
dimensional motion of a curve -faced object can be traced at a high speed and 
with accuracy, utilizing an object model, by applying as sensors a stereo 
camera system consisting of a group of image pick-up cameras. 
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What is claimed is: 

1 Object motion tracing method for tracing the three-dimensional 
motion of an object, based on the stereo image of the object concerned 
obtained in time series, composed of: 

a selecting step for selecting tracing points corresponding to the 
silhouette of the object concerned in the stereo image of the object; 

a sampling step for sampling from the above-mentioned stereo image 
the corresponding points on the silhouette of the object corresponding to 
respective tracing points; 

a measuring step for measuring the three-dimensional coordinates of 
the sampled corresponding points; and 

a detecting step for detecting the position/posture of the object from 
the three-dimensional coordinates of those respective tracing points and 
respective corresponding points, 

wherein 

the three-dimensional motion of aforesaid object is traced by 
continuously repeating each process from the selecting step through the 
detecting one toward each frame of the stereo image obtained in time 
series. 

2. Object motion tracing technique according to Claim (1), wherein 

the three-dimensional geometric model of an object is used to select the 
tracing points on the selecting step. 

3. Object motion tracing technique according to Claim (1) or (2), wherein 
the three-dimensional coordinates are measured by stereo vision applying 
stereo correspondence on the measuring step. 

4. Object motion tracing technique according to Claim (1) or (2), wherein 
the three-dimensional coordinates are measured by monocular vision on 
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the measuring step. 

5. A recording media wherein an object motion tracing program is 
recorded for tracing the three-dimensional motion of an object on the basis 
of the stereo image of the object obtained in time series, composed of: 

a selecting step for selecting the tracing point corresponding to the 
silhouette of the object concerned on the stereo image of the object; 

a sampling step for sampling from above-mentioned stereo image the 
corresponding points on the silhouette of the object corresponding to 
respective tracing points; 

a measuring step for measuring the three-dimensional coordinates 
for the sampled corresponding points; and 

a detecting step for detecting the position/posture of the object from 
respective three-dimensional coordinates of aforesaid tracing points and 
corresponding points, 
wherein 

the program for executing the tracing of the three-dimensional 
motion of the object by continuously repeating each process from the 
selecting step through the detecting one as mentioned above toward each 
frame of the stereo image obtained in time series. 

6. A recording media according to Claim (5) above, wherein the program 
is recorded for executing the selection of the tracing point using the 
three-dimensional geometric model of an object on the selecting step. 

7. A recording media according to Claim (5) or (6) above, wherein the 
program is recorded for executing the measurement of the three- 
dimensional coordinates by stereo vision applying stereo correspondence 
on the measuring step. 

8. A recording media according to above Claim (5) or (6), wherein the 
program is recorded for executing the measurement of the three- 
dimensional coordinates by monocular vision on the measuring step. 
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ABSTRACT OF THE DISCLOSURE 



After stereo image is inputted (SI), the tracing point corresponding to the 
silhouette of a curved object observed by the stereo image is selected on the 
basis of the information on the present position of the object (S2), using a 
three-dimensional geometric model. And the corresponding points on the 
silhouette corresponding to respective tracing points are sampled from the 
stereo image (S3), and their three-dimensional coordinates are measured (S4). 
Then, from the group of the three-dimensional coordinates of tracing points 
and corresponding points, the position/posture of the object concerned and the 
measurement error are determined (S5), to discriminate the error (S6). In 
case the error is not small enough, processes starting from (S2) are repeated, 
regarding the detected position/posture as a renewed present position, while, 
if the error is small enough, processes are repeated starting from (Si), 
regarding the detected position/posture as a renewed present position, 
thereby providing the technique for high-speed tracing of three-dimensional 
rigid body motion of an arbitrarily-curved and smooth faced object utilizing an 
object model, applying a stereo camera system as sensors. 
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System Configuration for Embodying the Present Invention 
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FIG.3 



An Example of Time-Series Stereo Image 
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An Example of Estimated Initial Position of a Curve-Faced Object 
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An Example of Net Configuration of a Curve-Faced Object Model 




FIG.6 

Method of Searching for a Corresponding Edge Point 
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Method of Searching for a Stereo Corresponding Point 
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FIG.8 

Method of Three-Dimensional Estimation of a Corresponding 
Point by Monocular Vision 
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Measurement Error of a Curve-Faced Object in Stereo Vision 




FIG. 10 



An Example of a Tracing Result of an Arbitrarily-Curved Object 
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listed below and have also identified below, by checking the box, any foreign application for patent or 
inventor's certificate or PCT International application having a filing date before that of the application 
on which priority is claimed. 

Prior Foreign Application(s) Priority Not Claimed 

2000-206818 Japan 7/July/2000 Q 

(Number) (Country) (Day/Month/Year Filed) 



(if applicable) 



□ 



(Number) 



(Country) 



(Day/Month/Year Filed) 



□ 



(Number) 



(Country) 



(Day/Month/Year Filed) 
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I hereby claim the benefit under 35 U.S.C. Section 119(e) of any United States provisional 
application(s) listed below: 



(Application Serial No.) 


(Filing Date) 


(Application Serial No.) 


(Filing Date) 


(Application Serial No.) 


(Filing Date) 



I hereby claim the benefit under 35 U. S. C. Section 120 of any United States application(s), or 
Section 365(c) of any PCT International application designating the United States, listed below and, 
insofar as the subject matter of each of the claims of this application is not disclosed in the prior 
United States or PCT International application in the manner provided by the first paragraph of 35 
U.S.C. Section 112, I acknowledge the duty to disclose to the United States Patent and Trademark 

. Office all information known to me to be material to patentability as defined in Title 37, C. F. R., 
Section 1 .56 which became available between the filing date of the prior application and the national 

_ or PCT International filing date of this application: 



(Application Serial No.) 


(Filing Date) 


(Status) 






(patented, pending, abandoned) 


(Application Serial No.) 


(Filing Date) 


(Status) 






(patented, pending, abandoned) 


(Application Serial No.) 


(Filing Date) 


(Status) 






(patented, pending, abandoned) 



I hereby declare that all statements made herein of my own knowledge are true and that all 
statements made on information and belief are believed to be true; and further that these statements 
were made with the knowledge that willful false statements and the like so made are punishable by 
fine or imprisonment, or both, under Section 1001 of Title 18 of the United States Code and that such 
willful false statements may jeopardize the validity of the application or any patent issued thereon. 
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POWER OF ATTORNEY: As a named inventor, I hereby appoint the following attorney(s) and/or 
agent(s) to prosecute this application and transact all business in the Patent and Trademark Office 
connected therewith, (list name and registration number) 

Ernest I. Gifford, P.O. Reg. No. 20,644 Avery N. Goldstein, P.O. Reg. No. 39,204 

Allen M. Krass, P.O. Reg. No. 1 8,277 Mark D. Schneider, P.O. Reg. No. 43,906 

Irvin L. Groh, P.O. Reg. No. 17,505 David R. Kurlandsky, P.O. Reg. No. 41,505 

Douglas W. Sprinkle, P.O. Reg. No. 27,394 

Thomas E. Anderson, P.O. Reg. No. 31,318 

Ronald W. Citkowski, P.O. Reg. No. 31,005 

Judith M. Riley, P.O. Reg. No. 31,561 

Douglas J. McEvoy, P.O. Reg. No. 34,385 

Ellen S. Cogen, P.O. Reg. No. 38,109 

Roberta J. Morris, P.O. Reg. No. 33,196 

John G. Posa, P.O. Reg. No. 37,424 



Send Correspondence to: Dou g ,as w - s P rinkl e 

Gifford, Krass, Groh, Sprinkle, Anderson & Citkowski, P. C. 
280 N. Old Woodward, Suite 400 
Jfi Birmingham, MI 48009 

0 1 Direct Telephone Calls to: (name and telephone number) 

Douglas W. Sprinkle (248) 647-6000 



fx 


Full name of sole or first inventor 
Yasushi SUMI 




Sole or first inventor's signature . . 






Date 

Auqust 21, 


2000 




Residence 1-103-503, Kasuga 1 


-choline , 


Tsukuba-shi , 


Ibara ki-ken , 


Japan 




Citizenship 
Japan 




Post Office Address 












1-103-503, Kasuga 1 


-chome , 


Tsukuba-shi , 


Ibaraki-ken , 


Japan 







Full name of second inventor, if any 
Fumiaki TOMITA 


Second inventor's signature 






Date 

August 21, 2000 


Residence 


3-776, 


Takezono, Tsukuba-shi, 


Ibara ki-ken , 


Japan 


Citizenship 
Japan 


Post Office Ad 


dress 

3-776, 


Takezono, Tsukuba-shi, 


Ibaraki-ken , 


Japan 
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Full name of third inventor, if any 
Yutaka ISHIYAMA 


ThirdiDventor'& signature „ /-> a 




Date 

August 21, 2000 


Residence 

2-1, Takanawa 3-chome, Minato-ku, Tokyo, 


Japan 




Citizenship 

Japan 


-- Post Office Address 

2-1, Takanawa 3-chome, Minato-ku, Tokyo, 


Japan 







Full name of fourth inventor, if any 



Fourth inventor's signature 



Citizenship 



Post Office Address 



Full name of fifth inventor, if any 



Fifth inventor's signature 



~ : Residence 



Post Office Address 



Full name of sixth inventor, if any 



Sixth inventor's signature 



Post Office Address 
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